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PREDICTING  JOB  DIFFICULTY  IN  HIGH  APTITUDE  CAREER  LADDERS 
WITH  STANDARD  SCORE  REGRESSION  EQUATIONS 


I.  INTRODUCTION 

Determining  the  relative  difficulty  of  jobs  has 
long  been  a desired  product  in  the  area  of  job 
evaluation  (Kelday,  1922).  Recently  a method  for 
developing  a comparative  measure  of  job  difficulty 
has  been  produced  (Lecznar,  1971;  Mead,  1970a, 
1970b;  Mead  & C^ristal,  1970).  The  method 
predicts  ranked  difficulty  of  whole  job  descrip- 
tions with  three  job  content  variables.  This 
capabQity  has  made  it  possible  to  undertake  Air 
Force  objectives  which  have  affected  enlisted  force 
personnel,  career  planning,  and  personnel  manage- 
ment research  (Christal,  1972,1974;  Wiley,  1972). 

The  methodological  studies  by  Mead  (1970a, 
1970b)  and  Mead  and  Christal  (1970)  used  equa- 
tions involving  task  difficulty  ratings  as  predictors 
to  account  for  the  rank  order  values  assigned  by 
noncommissioned  officers  (NCOs)  to  250  job 
descriptions  in  three  career  ladders.  These  ladders 
were  Medical  Materiel,  Accounting  and  Finance, 
and  Vehicle  Maintenance.  The  resulting  multiple 
Rs  were  .95  for  the  first  two  ladders  and  .93  for 
the  last.  Similarities  among  the  equations  indicate 
that  a single  common  equation  would  approximate 
the  job  difficulty  rankings  in  all  three  career 
ladders.  A standard  weiglit  equation  was  applied 
and  found  to  retain  the  R value  of  .95  for  the  first 
two  ladders  and  to  drop  to  .92  for  the  last. 

Subsequently,  ladder-specific  equations  were 
developed  in  nine  additional  career  ladders  and 
used  to  study  the  difficulty  of  jobs  performed  by 
first-term  airmen  (Wiley,  1972,  p.  4).  Although 
these  12  career  ladders  involved  jobs  of  differing 
composition  and  complexity,  the  same  three 
predictors  were  found  suitable.  An  alternate 
constant  standard  weight  equation  was  developed 
(based  on  the  12  ladders)  but  remained  to  be 
tested  in  more  technical  occupational  specialties 
(Christal,  1974,  p.  15).  Additional  studies 
involving  four  career ' ladders  were  considered 
necessary  before  extending  the  applications  of  the 
constant  standard  weight  equation. 

The  present  study  applies  and  expands  the  job 
difficulty  research  methodology  in  career  ladders 
which  require  entering  personnel  to  possess  high 
aptitude  in  the  general  and  the  electronics  areas. 


The  critical  question  asked  by  this  study  was:  Will 
a generalized  equation  for  job  difficulty  deter- 
mination apply  to  select  career  ladders  with 
electronics  or  general  aptitude  entrance  require- 
ments of  80  or  above?  The  specific  objectives  of 
the  investigation  were; 

1.  To  provide  equations  for  evaluating  jobs 
which  are  based  on  three  job  content  variables  for 
the  Aircraft  Control  and  Warning  (AC&W)  Radar, 
Ground  Radio  Communications,  Weather 
Observer,  and  Information  Specialist  career 
ladders. 

2.  To  test  the  predictive  effectiveness  of  the 
constant  standard  weight  equations  when  applied 
in  these  four  career  ladders. 

3.  To  compare  the  predictive  effectiveness  of 
the  constant  standard  weight  equations  with  that 
of  each  specific  ladder  equation. 


II.  .METHOD 

Job  and  task  difficulty  measures  were 
developed  using  various  features  of  other  job 
evaluation  procedures  (Christal,  1967;  Mead  & 
Christal,  1970).  Job  descriptions  were  selected  and 
arranged,  then  ranked  in  varying  random  order 
contexts  by  supervisory  personnel  to  develop  job 
difficulty  criterion  measures.  Task  difficulty 
ratings  were  obtained  to  develop  predictor  vari- 
ables being  examined  ,by  multiple  regression 
analysis.  An  analysis  design  for  testing  the 
predictive  effectiveness  of  various  constant 
standard  weight  equations  (including  the  Mead  and 
Christal  equation)  was  develof>ed  in  keeping  with 
objective  3 given  previously. 

Selection  and  Arrangement 
of  Job  Descriptions 

Two-hundred  fifty  job  descriptions  were 
randomly  selected  from  all  jobs  identified  in  each 
of  the  four  career  ladder  occupational  surveys  as 
given  in  Table  1.  To  control  for  context  effects, 
job  descriptions  were  randomly  placed  in  varying 
contexts.  Sixteen  different  contexts  were 
developed,  each  containing  ten,  25-job-description 
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^ Table  I.  Population  and  Survey  Characteristics 


CarMr 

Laddart 

Number 
of  Jobs 
Survey  ad 

Number 
Tacict  In 
inventory 

Aptitude 

Entrance 

Requirement 

AC&W  Radar 

1,375 

456 

Electronics  80 

Ground  Radio  Communications 

2,112 

326 

Electronics  80 

Weather  Observer 

1,549 

441 

General  80 

Information  Specialist 

616 

388 

General  80 

subsets  using  the  Individual  Job  Description 
(JOBIND)  sample  selection  routine  of  the  Compre- 
hensive Occupational  Data  Analysis  Programs 
(CODAP),  developed  by  Stacey,  Weissmuller, 
Barton,  and  Rogers  (1974). 

Each  job  description  was  printed  for  distrib- 
ution in  the  above  arrangement.  The  printed  job 
descriptions  each  contained  an  identification 
number,  a list  of  tasks,  the  percent  of  time  job 
incumbents  spent  performing  each,  and  the 
cumulative  time  spent,  under  respective  columnar 
headings. 

Development  of  Job 
Difficulty  Criterion  Measures 

One-hundred  sixty  7-skill  level  and  9-skill  level 
noncommissioned  officers  (NCO)  working  in  each 
career  ladder  were  requested  to  evaluate  the 
relative  difficulty  of  the  job  descriptions  of  the 
respective  career  ladders.  This  request  was  made 
by  mail  through  consolidated  base  personnel  office 
(CBPO)  persormel  who  were  requested  to  select 
the  desired  number  of  NCOs. 

Each  of  the  640  individuals  was  provided  a 
packet  of  materials  which  included  one  subset  of 
25  job  descriptions,  and  instruction  sheet,  a ranker 
background  information  sheet,  and  a job  ranking 
sheet.  The  NCOs  were  requested  to  carefully 
review  each  of  the  25  job  descriptions  and  rank 
order  them  according  to  relative  difficulty. 

Development  of  Task 
Difficulty  Measures 

A job  inventory  containing  all  tasks  (see  Table 
1)  for  each  career  ladder  was  constructed;  each 
inventory  contained  a relative  difficulty  rating 
scale.  The  respective  inventory  booklets  were 
printed  with  the  7-point  relative  difficulty  scale  at 


the  top  of  each  page  for  ready  rater  reference. 
Task  difficulty  was  defined  as  time  required  to 
learn  to  do  the  task  satisfactorily. 

Personnel  with  7-  and  9-skill  level  Air  Force 
specialty  codes  (AFSC)  in  these  four  career  ladders 
provided  ratings  on  relative  difficulty  of  the  task 
statements.  These  personnel  completed  the 
inventories  under  the  direction  of  CBPOs. 

Development  of  Predictor  Variables 

In  keeping  with  the  Mead  and  Christal  (1970) 
methodological  study,  three  predictors  were 
developed  in  all  four  ladders  based  on  the 
assumption  that  they  would  yield  efficient  predic- 
tion. Predictor  1 was  the  number  of  tasks  in  the 
job,  and  predictor  2 was  the  number  of  tasks 
squared.  Inclusion  of  the  squared  term  was 
necessary  because  Mead  and  Christal  (1970)  found 
a curvilinear  relationship  between  the  number  of 
tasks  in  jobs  and  their  perceived  difficulty  levels. 
Predictor  3 was  a complex  variable  reflecting  the 
average  difficulty  level  of  tasks  performed  per  unit 
time  spent.  This  variable  was  computed  as  simply 
the  cross-products  of  time  spent  and  task 
difficulty,  summed  across  ail  tasks  in  the  inventory 
for  a particular  job.  Additional  variables  were 
developed  in  the  electronics  ladders  using  task 
difficulty  measures. 

Analysis  of  Supervisors' 

Judgment  Policy 

The  predictor  variables  were  analyzed  by 
multiple  regression  to  capture  the  supervisors’  job 
difficulty  evaluation  policy.  These  problems  were 
hypothesized  as  containing  the  most  logical 
grouping  of  predictors  for  the  current  study  on  the 
basis  of  the  earlier  results.  For  each  problem,  R 
and  values  were  derived. 
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^Examined  in  electronics  ladders  only. 
^Intraclass  correlation  cofficient. 
^'SpearinanOBrown  correlation  cofficient. 


Supervisor  agreement  concerning  the  relative 
difficulty  of  jobs  ranked  varied  in  the  four  ladders. 
However,  the  reliability  estimates  approximate 
those  obtained  in  the  12  other  ladders  (Mead. 
1970a,  1970b;  Mead  & Christal,  1970;  Wiley, 
1972).  These  findings  appear  to  warrant  the  use  of 
multiple  regression  analysis  to  capture  the 
supervisor  ranking  policies. 


Thereafter  a refining  process  was  applied  in  the 
two  electronics  ladders  to  the  criterion  measures. 
In  this  process  each  rank  value  for  each  of  the  2.‘i 
jobs  evaluated  by  a supervisor  was  correlated  with 
the  mean  rank  value  obtained  from  all  supervisors. 
In  turn,  a t test  of  significance  was  applied  to 
determine  if  any  supervisor  judgment  deviated 
significantly  from  the  mean  rank  values.  Cases 
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with  nonsignificant  t values  (N  = 7 in  the  Ground 
Radio  Coininuiiications  ladder  and  N = 25  in  the 
AC&W  Radar  ladder)  were  removed;  then  the 
criterion  measures  and  reliability  estimates  were 
recomputed.  With  the  rank  values  for  those  cases 
removed,  additional  interrater  reliability  estimates 
were  computed.  Table  2 reports  these  much 
improved  Rj  j and  Rtt  values.  As  indicated,  a 
refined  form  of  the  criterion  measure  was 
developed  for  the  AC&W  Radar  and  Ground  Radio 
Communications  ladders.  These  criteria  are 
considered  in  the  following  problems. 


Task  Difficulty  Ratings 

A difficulty  value  was  computed  for  each  task 
from  rating  bixrklets  returned  by  supervisors  in  the 
four  career  ladders.  The  intraclass  correlation 
technique  in  tlie  CODAP  RK.XALL  program 
(Stacey  et  al.,  1974)  was  used  to  compute  the 
estimates  of  reliability.  These  estimates  are 
presented  in  Table  3.  Tire  reliability  estimates 
obtained  for  each  ladder  were  quite  similar.  In  the 
electronics  ladders,  a noticeable  improvement  in 
the  Ri  1 value  was  obtained  for  the  total  group  by 


Table  3.  Estimated  Reliability 
of  Task  Difficulty  Measures 


Career  Ladder 

K 

•’kx 

AC&W  Radar 

82 

.28 

.97 

AC&W  Radjir  (reduced  N) 

78 

.33 

.98 

Ground  Radio  Communications 

94 

.25 

.97 

Ground  Radio  Communication  (reduced  N) 

89 

.30 

.97 

Weather  Observer 

78 

.26 

.96 

Information  Specialist 

54 

.24 

.94 

removing  a number  of  disparate  cases  which  did 
not  correlate  appreciably  with  the  mean  task  values 
for  all  raters.  Using  the  definition  of  task  difficulty 
and  the  7-point  scale  employed  in  these  studies, 
results  indicate  that  NCOs  provide  stable  measures. 

Development  of  Job  Difficulty 
Prediction  Equations 

Three  predictor  variables  were  analyzed 
concerning  their  potential  for  use  in  a regression 
equation  to  provide  job  difficulty  values  similar  to 
the  supervisors’  ranking  policies.  The  three 
predictors  obtained  by  Mead  (i  970a,  1970b)  were 
found  to  be  efficient  in  these  regression  equations. 
The  correlation  between  the  derived  and  criterion 
job  difficulty  values  yielded:  R^  = .86  for  AC&W 
Radar;  R^  = .93  for  Ground  Radio  Communica- 
tions; R^  = .79  for  Weather  Observer;  and  R’  = 
.86  for  the  Information  Specialist  career  ladder. 

Resulting  equations  are  given  in  Table  4 with 
those  equations  obtained  in  the  12  other  ladders. 
The  standard  score  weights  for  the  predictor  vari- 
ables are  reasonably  uniform  over  all  16  ladders. 


IV.  DEVKLOPMEVr  AND  ANALYSIS 
OF  CONSTANT  STANDARD  WEIGHT  EOLATION 

Various  constant  standard  weight  equations 
were  developed  and  tested  for  predictive  effective- 
ness in  the  AC&W  Radar  and  Ground  Radio 
Communications  career  ladders.  In  this  process 
predicted  job  difficulty  values  were  computed 
using  each  constant  standard  weiglit  equation.  The 
predicted  values  obtained  for  each  equation  were 
then  correlated  with  the  criterion  measure  and 
tested  for  significant  differences  using  the  F 
statistic.  A constant  standard  weiglit  equation  was 
selected  based  on  the  results. 

/ 

The  standard  score  weiglits  for  the  tliree 
variables  were  drawn  from  each  ladder-specific 
equation  (as  required  to  create  a particular 
constant  standard  weight  equation)  and  mean 
standard  score  weights  were  successively 
computed.  The  mean  standard  score  weights  (for 
four  constant  standard  weight  equations)  were 
applied  to  the  predictor  data  from  each  of  the  two 
career  ladders  to  derive  job  difficulty  indices  (JDf) 
using  the  formula. 
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Table  4.  Predictors  and  Associated  Standard  Score  (SS)  Weights 


Squared  Average  Task 

Number  Number  Difficulty 

Career  of  Tasks  of  Tasks  per  Unit 

Ladder  Performed  f^rformed  Time  Spent 


1.  Vehicle  Maintenance 

2.  Medical  Materiel 

3.  Accounting  and  Finance 

Mead  and  Christal  SS  Equation 

4.  Security  Police 

5.  Administration 

6.  Materiel  Facilities 

7.  hventory  Management 

8.  Fuel  Services 

9.  Transportation 

10.  Fire  Protection 

11.  CE  Structural/Pavements 

12.  Electrical  Power  Production 

SS  Equation  for  12  Career  Ladders 

13.  AC&W  Radar 

14.  Ground  Radio  Communications 

15.  Weather  Observer 

16.  Information  Specialist 

SS  Equation  for  16  Career  Ladders 


1.2913 

0.6153 

0.5161 

1.1258 

-0.5867 

0.4526 

1.5851 

-fi.9584 

0J923 

1.3341 

-0.7201 

0.45c(7 

1.2152 

-0.6925 

0.4901 

1.5143 

-0.7825 

0.2427 

1 .4724 

-0.8707 

0.3310 

1.8287 

-1.1442 

0.2236 

1 .3686 

-0.8188 

0.4967 

1.6957 

-1.0797 

0.3039 

0.9389 

-0.5065 

0.6659 

1 3874 

-0.7706 

0.2806 

1.6607 

-0.9411 

0.2059 

1 .4237 

-0.8139 

0.3834 

1.8139 

-1.0511 

0.4814 

1 .4824 

-0.7789 

0.2479 

0.9525 

-0.3484 

0.7020 

1.7444 

-0.9732 

0.1750 

1.4431 

-0.8236 

0.4010 

PlOcX,  - X]  020^X2  - Xj  020^X2  - X3 

Y= + + + C, 

0|  ^2  ^3 

where 

Y = job  difficulty  index  (JDI) 

Oc  = standard  deviation  of  criterion  job 

difficulty 

Xi  = number  of  tasks  performed  (NTSK) 

X,  = mean  number  of  tasks  pierformed 

(3i  = constant  standard  score  weight  for 

X, 

Oi  = standard  deviation  of  X, 

Xj  = squared  number  of  tasks  performed 

(SNTSK) 

Xj  = mean  squared  number  of  tasks 

performed 

(Jj  = constant  standard  score  weight  for 

X, 

Oj  = standard  deviation  of  Xj 


average  task  difficulty  per  unit  time 
spent  (ATDPUT) 

mean  average  task  difficulty  per 
unit  time  sfient 

constant  standard  score  weight  for 

X, 

standard  deviation  of  X3 

mean  criterion  job  difficulty  value 
obtained  by  combining  the  ranked 
scores  for  each  ladder  involved. 
These  values  were  obtained  with  a 
McNemar  technique  (1962,  p.  18). 

Each  set  of  the  JDls  computed  with  the 
constant  standard  weight  equations  permitted  a 
comparison  and  a possible  maximization  of  the 
correlations  between  the  predicted  values  and 
supervisors’  evaluations.  These  correlations  and 
associated  F tests  are  reported  in  Table  5.  They 
indicate  that  each  constant  standard  weight 
equation  approximated  the  predictive  efficiency  of 
the  ladder-specific  equations.  As  far  as  the 
variances  are  concerned,  any  one  of  the  equations 


X3  = 
X3  = 

P3  = 

02  = 
c 
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Table  5.  Multiple  Correlations  and  F Ratios*  Comparing 
Specific  Versus  Generalized  Equations 


Equations 

AC&W  Radar 

Ground  Radio  Communications 

R* 

R 

F ratio 

R^ 

R 

F ratio 

Ladder-Specific  Equations 

.698 

.836 

.825 

.909 

Three-Ladder  Equation 

.688 

.829 

1.03 

.788 

.888 

1.21 

Twelve-Ladder  Ekjuation 

.689 

.830 

1.03 

.803 

.896 

1.12 

Two-Ladder  Electronics  Equation 

.689 

.829 

1.03 

.819 

.904 

1.04 

Sixteen-Ladder  Equation 

.698 

.835 

1.00 

.807 

.898 

1 10 

^Nonsignificant  at  the  .02  level  for  tlf  - 249.  F ratios  were  obtained  using  Ciui! ford’s  (1956)  formula  1 0.1 1 . larger 
variance/sinaller  variance  (1  ~ 


could  have  represented  tlie  populations  almost  as 
well  as  the  ladder-specific  equations. 

In  Table  5,  it  is  apparent  that  there  would  be 
no  advantage  to  develop  separate  constant 
standard  weiglit  equations  for  each  aptitude 
entrance  requirement  area.  The  R^s  and  F ratios 
for  the  two-ladder  electronics  equations  were  no 
better  than  the  12-  or  the  16-ladder  equations. 

To  further  insure  that  some  other  combination 
of  career  ladders  would  not  produce  a better 
constant  standard  weight  equation,  the  16  ladder- 
specific  equations  (given  in  Table  4)  were 
hierarchically  grouped  using  the  Bottenberg  and 
Christa!  (1961)  technique.  The  results  indicated 
that  five  groups  of  career  ladders  (equations) 
would  be  required  to  increase  the  overall  R^  by 


.03  from  .84.  This  small  increase  in  predictive 
efficiency  made  it  impractical  to  consider  pursuing 
a separate  equation  for  each  aptitude  area. 

Tlierefore,  it  was  of  interest  to  closely  compare 
the  predictive  effectiveness  of  the  1 2-  and  the 
16-ladder  constant  standard  weight  equations  witli 
file  ladder-specific  (or  least  squares)  equation  for 
each  ladder.  The  multiple  correlations  (Table  6) 
for  the  12- and  the  16-ladder  eq'.  jtions  are  in  close 
agreement.  They  also  compare  favorably  with  the 
multiple  correlations  for  the  ladder-specific  (least 
squares)  equations. 

It  is  interesting  fiiat  the  original  1 2-ladder  equa- 
tion held  up  when  it  was  applied  to  each  of  the  16 
ladders.  Only  practical  considerations  would 
suggest  that  the  16-ladder  equation  should  be  used 
over  the  1 2-ladder  equation. 


Table  6.  Ladder  by  Ladder  Comparisons  of  Least  Squares 
Equations  with  Two  Standard  Weight  Equations 


ISAFSC  12  AFSC 

Leift  Mn  SS  MnSS 

Squares  Weights  Weights 

Career  Ladders  R R R 


811X0  Security  Police 

702X0  Administrative 

647X0  Supply  Materiel  Facilities 

645X0  Inventory  Management 

631X0  Fuel  Services 

605X0  Transportation 

571X0  Fire  Portection 

551X0  CE  Structural/Pavements 

543X0  Electrical  Power  Production 

473X0  Vehicle  Maintenance 

915X0  Medical  Materiel 

67 1 XO  Accounting  & Finance 

252X0  Weather  Observer 

791X0  Information  Specialist 

303X0  AC&W  Radar 

304X4  Ground  Radio  Communications 


.9215 

.9132 

.9138 

.9772 

.9714 

.9705 

.9417 

.9402 

.9404 

.9358 

.9173 

.9170 

.9420 

.9349 

.9359 

.9305 

.9229 

.9235 

.9390 

.8867 

.8879 

.9285 

.9264 

.9260 

.9374 

.9235 

.9226 

.9269 

.9158 

.9153 

.9487 

.9450 

.9450 

.9511 

.9499 

.9503 

.8870 

.7966 

.7972 

.9296 

.8983 

.8964 

.8356 

.8353 

.8355 

.9086 

.8684 

.8970 

V ( ()N(  1 I SIONS  and  IMl*l.K  ATK)NS 

Joh  aiul  lask  dilticuliy  measures  were 
(Je\eli)peil  in  the  tour  career  ladders  studied,  llie 
suiXTvisor  job  evaluation  policies  were  captured  in 
each  ladder  in  a three-variable  multiple  regression 
eciuation.  Hie  results  from  the  four  career  ladders 
added  by  this  study  were  considered  sulTiciently 
mt'icient  when  grouped  with  results  obtained  in  1 2 
other  career  ladders  to  develop  a comprehensive 
constant  standard  weiglit  equation.  Tliis  compre- 
hensive equation  was  based  upon  standard  score 
weights  developed  in  lb  career  ladders  representa- 
tive of  the  four  aptitude  areas  (General, 
Administrative,  Mechanical,  and  Electronics) 
required  for  entry  into  Air  Force  occupational 
specialties.  From  this  standpoint,  the  16-ladder 
equation  is  more  comprehensive  (inclusive)  than 
the  1 2-ladder  equation.  Other  results  indicated 
that  equations  produced  by  grouping  ladders  from 
like  aptitude  areas  were  no  more  efficient  than  the 
generalized  equations. 


Hie  vaiiance  which  has  been  captured  relative 
to  the  job  dittlcuity  criterion  measures  may  he 
attributed  to  the  observability  of  a distinction  in 
performance,  lire  task  difficulty  measures  permit 
a primary  technique  for  observing  this  distinction. 

The  results  sliow  little  (if  any)  improvement 
was  gained  by  using  the  164adder  equation  instead 
of  the  12-ladder  equation  to  predict  the  job 
difficulty  policy  rankings.  However,  from  a 
practical  standpoint,  it  would  appear  that  the 
comprehensive  constant  weight  equation 
developed  for  16  career  ladders  representing  all 
four  aptitude  areas  would  be  more  generalizable  to 
all  Air  Force  career  ladders  than  the  equation 
based  upon  1 2 career  ladders  representing  two 
aptitude  areas. 

Tlie  12-ladder  constant  standard  weight  equa- 
tion has  been  used  to  operationally  derive  diffi- 
culty indices  for  jobs  surveyed  in  all  the  Air  Force 
career  ladders  which  have  been  studied.  In  the 
future,  it  would  appear  advantageous  to  use  the 
16-ladder  equation  instead  of  the  12-ladder 
equation. 
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